Papers with contextualized embedding
Syntax-aware Multi-task Graph Convolutional Networks for Biomedical Relation Extraction (D19-62)
Copied to clipboard
| Challenge: | 80% of the data sets for relation extraction tasks are negative instances, resulting in a lack of syntactic information between two entity mentions. |
| Approach: | They propose a graph convolutional networks model that incorporates dependency parsing and contextualized embedding to capture comprehensive contextual information. |
| Outcome: | The proposed model achieves state-of-the-art F-score on the 2013 drug-drug interaction extraction task. |
Quantifying the Contextualization of Word Representations with Semantic Class Probing (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Pretrained language models are effective in solving NLP tasks, but there are still questions about how and why they work so well. |
| Approach: | They use BERT to quantify contextualization by studying the extent of inference . they show that top layer representations support highly accurate inference of semantic classes . |
| Outcome: | The proposed model is highly accurate, but weak in the lower layers . it is more task-specific after finetuning while lower layers are more transferable . |
Connecting the Dots: Inferring Patent Phrase Similarity with Retrieved Phrase Graphs (2024.findings-naacl)
Copied to clipboard
| Challenge: | Existing methods for inferring patent phrase similarity do not perform satisfactorily . et al., 2010: patents are pivotal to innovation, safeguarding novel ideas . |
| Approach: | They propose a graph-augmented approach to amplify patent phrase contextual information . they construct a phrase graph that links to patents cited by or cited in patents for each phrase . |
| Outcome: | The proposed approach significantly improves the representation of patent phrases in a self-supervised fashion. |
DirectProbe: Studying Representations without Classifiers (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches for probing opaque representations often use training classifiers and use the accuracy, mutual information, or complexity as a proxy for the representation’s goodness. |
| Approach: | They propose a heuristic that directly studies the geometry of a representation by building upon the notion of 'version space' they argue that doing so can be unreliable because different representations may need different classifiers . |
| Outcome: | Experiments with linguistic tasks and contextualized embeddings show that even without training classifiers, DirectProbe can shine lights on how an embeddable space represents labels and anticipate the classifier performance for the representation. |
Unsupervised Hallucination Detection by Inspecting Reasoning Processes (2025.emnlp-main)
Copied to clipboard
| Challenge: | Unsupervised hallucination detection aims to identify hallucines generated by large language models without relying on labeled data. |
| Approach: | They propose an unsupervised method to detect hallucinated content by large language models . they use internal representations intrinsic to factual correctness to prompt the model to verify the truthfulness of a given statement . |
| Outcome: | The proposed framework outperforms existing unsupervised methods and is fully unsupervised and low cost. |